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conrimon  theme  within  today's 
Department  of  Defense  (DoD)  acqui¬ 
sition  community  is  the  importance  of 
compaition  in  reducing  technical  and 
cost  risik,  and  in  ensuring  that  a  pro¬ 
gram's  t^hnology  solution  is  mature 
enough  based  on  where  the  program 
is  located  within  the  acquisition  frame¬ 
work.  To  emphasize  how  foundational 
the  concept  of  competition  is  in  today's 
acquisition  environment,  a  program's 
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ability  to  "promote  real  competition"  is  one  of  the  five  major 
areas  comprising  DoD's  Better  Buying  Power  initiative  identi¬ 
fied  to  improve  organizational  and  program  efficiencies. 

While  most  acquisition  professionals  feel,  fundamentally,  that 
"competition"  is  important,  what  are  the  positives  and  chal¬ 
lenges  of  adding  competition  as  part  of  a  competitive  prototyp¬ 
ing  process  in  support  of  an  acquisition  strategy?  To  answer 
this  question,  we  will  look  at  the  competition  process  from 
a  sitting  program  manager's  (PM's)  perspective  and  discuss 
the  aspects  and  impacts  of  a  competitive  prototyping  process 
during  a  program's  Technology  Development  (TD)  phase. 

Our  specific  example  is  the  Joint  and  Allied  Threat  Aware¬ 
ness  System  (JATAS-AN/AAR-59)  program  within  Program 
Management  Aviation-272  (PMA-272)  (Advanced  Tactical 
Aircraft  Protection  Systems),  part  of  the  Naval  Air  Systems 
Command  (NAVAIR)  Program  Executive  Office  for  Tactical 
Aircraft  (PEO-T)  at  Naval  Air  Station  Patuxent  River,  Md.  The 
AN/AAR-59  is  an  Acquisition  Category  (ACAT)  1C  program 
that  recently  completed  a  competitive  prototyping  TD  phase 
between  two  contractors.  The  competitive  prototyping  pro¬ 
cess  resulted  in  a  Milestone  B  decision  and  subsequent  Engi¬ 
neering  and  Manufacturing  Development  (EMD)  acquisition 
phase  contract  being  awarded  in  early  third-quarter  Fiscal  Year 
2011.  The  AN/AAR-59's  Initial  Operating  Capability  (IOC) 
date  is  2015  onboard  the  MV-22  Osprey  aircraft  platform. 

In  the  Beginning 

The  Counter/Counter  Air  Defense  Initial  Capabilities  Docu¬ 
ment  (ICD)  dated  June  15,  2006,  established  the  need  to  in¬ 
crease  the  survivability  of  assault  aircraft  operating  in  hostile 
environments.  In  Fiscal  Year  2007,  the  Chief  of  Naval  Opera¬ 
tions  Air  Warfare  Division  (OPNAV)  N98  (Air  Warfare  Divi¬ 
sion)  executed  an  independent  Analysis  of  Alternatives  (AoA) 
to  evaluate  alternatives  to  meet  the  capability  gap  identified  in 
the  Counter/Counter  Air  Defense  ICD.  The  AoA  results  deter¬ 
mined  that  threat-warning  technology  was  mature  enough  to 
proceed  with  an  advanced  Missile  Warning  System  (MWS). 
Subsequently,  OPNAV/N98  drafted  a  Capabilities  Devel¬ 
opment  Document  that  designated  the  system  as  the  AN/ 
AAR-59.  The  Department  of  the  Navy  approved  the  draft 
document  to  enter  the  Joint  Requirements  Oversight  Council 
review  process. 

The  JATAS  team  spent  the  summer  of  2007  preparing  pro¬ 
gram  documentation  for  an  intended  program  initiation  at 
Milestone  B.  In  September  and  November  2007,  the  Under 
Secretary  of  Defense  for  Acquisition,  Technology  and  Logis¬ 
tics  (USD[AT&L])  and  the  Assistant  Secretary  of  the  Navy 
for  Research,  Development  and  Acquisition  (ASNfRDA]) 
released  memos  on  prototyping  and  competition.  Through¬ 
out  the  spring  of  2008,  the  program  worked  with  PEO-T, 
ASN(RDA),  and  Deputy  Under  Secretary  of  Defense  for 
Portfolio  Systems  Acquisition  to  design  the  architecture  of 
a  competitive  prototyping  Technology  Development  Strat¬ 
egy  (TDS).  By  the  time  the  updated  DoDI  5000.02  was  re¬ 


leased  in  late  2008,  including  a  requirement  for  competitive 
prototyping,  the  program  was  already  well  under  way  with 
documentation  and  contracting  plans  for  a  competitive  TD 
phase.  The  JATAS  TDS  was  signed  in  December  2008,  and 
in  January  2009  the  final  Request  for  Proposal  (REP)  for 
a  competitive  TD  phase  was  released.  In  April  2009,  then 
USD(AT&L)  John  J.  Young  Jr.,  concerned  about  the  possibil¬ 
ity  of  uncoordinated  missile  warning  and  countermeasure 
approaches  on  similar  systems  by  each  of  the  Services,  is¬ 
sued  an  Aircraft  Survivability  Equipment  (ASE)  Acquisition 
Decision  Memorandum  (ADM).  This  memorandum  directed 
a  more  coordinated  solution  for  missile  warning  and  counter¬ 
measures.  The  PMA-272  JATAS  program  was  designated  an 
Acquisition  Category  1  (ACAT  1C)  special  interest  program, 
and  the  ASN(RDA)  was  designated  as  the  Milestone  De¬ 
cision  Authority.  The  ADM  effectively  endorsed  the  Navy 
Program  Management  Office  use  of  competitive  prototyping 
as  part  of  its  acquisition  strategy. 

TD  Phase:  the  Heart  of  Competitive 
Prototyping 

The  JATAS  program's  acquisition  strategy  determined  that  a 
16-month  competitive  prototyping  TD  phase  would  be  used 
that  included  two  prime  contractors:  Alliant  Techsystems  and 
Lockheed  Martin.  The  effort  would  be  managed  via  a  cost- 
plus,  incentive-fee  contract.  Each  prime  contractor  completed 
a  System  Requirements  Review,  a  System  Functional  Review, 
and  a  Preliminary  Design  Review  that  resulted  in  an  approved 
allocated  baseline  for  its  respective  AN/AAR-59  design.  In 
addition,  both  contractors  completed  prototype  ground  and 
flight  tests,  and  modeling  and  simulation  were  used  to  predict 
system  performance. 

JATAS's  implementation  of  a  competitive  prototyping  phase 
resulted  in  some  intended  and  unintended  consequences.  To 
capture  the  positives  and  challenges  that  PMA-272  experi¬ 
enced  as  an  outcome  of  implementing  a  competitive  proto¬ 
typing  phase,  various  program  stakeholders  were  interviewed 
and  program  documentation  analyzed  to  gain  insight  into  the 
PMA-272  competitive  prototyping  phase.  This  resulted  in  a 
white  paper  in  November  2010  that  identified,  from  the  gov¬ 
ernment's  perspective,  positives  and  challenges  with  respect 
to  TD  competitive  prototyping  during  the  AN/AAR-59  pro¬ 
gram.  A  synopsis  of  these  findings  follows  starting  with  the 
positives. 

Positives 

•  Responsive  Contractors:  Though  the  program  was  not 
technically  in  a  source  selection  environment  for  a  TD  con¬ 
tract  execution,  the  arrangement  resulted  in  a  "competi¬ 
tive  environment"  additionally  fostered  by  the  government 
team.  Both  TD  contractors  were  extremely  responsive  to 
the  government  during  TD  contract  execution,  allowing 
the  program  to  maintain  its  aggressive  schedule.  Both  con¬ 
tractors  were  reluctant  to  exceed  planned  costs,  even  on 
cost-plus,  incentive-fee  contracts,  because  of  the  percep¬ 
tion  of  competition.  The  performance  of  each  contractor  in 
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cost,  schedule,  and  performance  was  to  be  an  important 
discriminator  in  the  next  phase  of  the  competition. 


•  Competitive  Future  Pricing:  As  a  result  of  a  competi¬ 
tive  prototyping  effort,  the  program  was  able  to  award 
a  Fixed-Price  Incentive  Firm  (FPIF)  contract  for  the 
EMD  phase,  an  FPIF  option  for  Low-Rate  Initial  Pro¬ 
duction,  and  Firm-Fixed  Price  (FFP)  options  for  the 
first  seven  full-rate  production  lots,  as  well  as  FFP 
options  to  purchase  hardware  and  software  data 
rights  to  enable  future  competition.  Prices  were 
formulated  by  contractors  in  a  competitive  envi¬ 
ronment,  resulting  in  the  lowest  possible  cost  to 
the  government.  Competing  for  production  contract 
awards  and  purchasing  the  data  rights  are  not  typically 
affordable  during  a  TD  program. 
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•  Technical  Risk  Reduction:  Prototyping  during  the  TD  phase 
reduced  technical  risk  for  the  AN/AAR-59  program.  Both 
contractors,  as  a  result  of  competition,  made  significant  ef¬ 
forts  toward  early  integration  of  their  designs.  Early  looks 
at  hardware  before  Preliminary  Design  Reviews  (and  analy¬ 
sis  based  on  data  collected  during  government  prototype 
testing)  increased  government  confidence  in  contractor 
assertions  of  predicted  performance  of  the  EMD  designs. 
Prototype  data  from  two  separate  approaches  also  allowed 
the  AN/AAR-59  program  to  more  accurately  evaluate  Tech¬ 
nology  Readiness  Level.  With  all  system  performance  being 
equal,  the  EMD  and  production  contracts  were  able  to  be 
awarded  based  on  total  cost. 

•  Program  Execution  Risk  Reduction:  Dual  contract  execu¬ 
tion  in  a  competitive  environment  allowed  the  government 
team  to  observe  in  real  time  the  effectiveness  of  the  cor¬ 
porate  management  systems,  earned  value  performance, 
program  management,  and  contract  execution  of  the  TD 
contractors.  The  government  team  was  able  to  leverage  this 
experience  and  insight  to  assess  program  execution  risk  for 
each  contractor  during  the  EMD  source  selection.  Personal 
relationships  were  developed  between  the  contractor  and 
government  teams,  reducingthe  time  required  during  EMD 
for  the  joint  team  to  become  effective.  As  a  result,  the  teams 
developed  a  clearer  understanding  of  areas  that  led  to  effec¬ 
tive  communication  paths,  roles,  and  responsibilities. 

Additionally,  both  the  AN/AAR-59  system  and  the  govern¬ 
ment's  technical  team  matured  during  the  TD  phase.  The  tech¬ 
nical  team  had  a  chance  to  observe  two  technical  approaches, 
exposing  team  members  to  greater  technical  insight  compared 
with  monitoring  a  single  development  phase.  As  a  result,  the 
government  team  members  felt  they  would  be  more  effective 
during  EMD  because  of  this  experience. 

Also,  a  set  of  documentation  core  to  the  program  was  devel¬ 
oped  during  the  TD  phase.  These  core  program  documents 
then  could  be  leveraged  into  the  development  of  other  prod¬ 
ucts  such  as  the  Acquisition  Strategy,  Systems  Engineering 


development  program. 


Plan  and  the  Test  and  Evaluation  Master  Plan  supporting  the 
EMD  phase.  Finally,  there  was  the  opportunity  to  collect  sen¬ 
sor  data  to  support  development  of  new  algorithms.  These 
data  would  have  been  required  for  completion  of  EMD,  but 
waiting  to  collect  them  during  EMD  would  have  introduced 
additional  technical  and  schedule  risk  much  later  into  the  pro¬ 
gram's  acquisition  life  cycle. 

Chcdlenges 

While  there  were  positives  associated  with  PM  A-272's  JATAS 
competitive  prototyping  efforts  during  the  TD  phase,  there 
also  were  some  program  challenges. 

•  Government  Workload:  Administering  two  TD  contracts 
increased  the  government  workload  without  a  compara¬ 
ble  increase  in  team  size,  doubling  the  meetings,  Contract 
Data  Requirements  Lists  to  review,  and  contract  admin¬ 
istration.  The  AN/AAR-59  government  team  executed 
eight  major  reviews  (two  each  of  Integrated  Baseline  Re¬ 
views,  System  Requirement  Reviews,  System  Functional 
Reviews,  and  Preliminary  Design  Reviews— instead  of 
one  each)  in  13  months,  while  simultaneously  prepar¬ 
ing  for  EMD  source  selection  and  a  Milestone  B  review. 
In  addition,  the  Systems  Engineering  Technical  Review 
events  doubled  the  attendance  requirements  for  senior 
engineering  and  logistics  competency  members  to  sit  as 
board  members  and  provide  subject  matter  expertise. 

Acase  could  be  madeforthe  necessity  of  three  teams  suc¬ 
cessfully  executing  a  competitive  prototyping  effort.  To  pre¬ 
vent  the  government  from  inadvertently  leveling  the  techni¬ 
cal  solution  by  having  the  same  government  team  deal  with 
both  contractors,  the  AN/AAR-59  Logistics  and  Engineering 
disciplines  built  two  teams  to  interface  directly  with  each  of 
the  two  vendors.  The  need  for  a  third  team  immediately  was 
evident  to  provide  the  link  between  the  two,  prepare  for  the 
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staying  abreast,  and, 
in  some  cases,  ahead  of 
emerging  policy  guidance 
can  help  a  program  team 
make  progress  in  a  changing 
environment. 


were  complete  and  the  Capabilities  Development  Docu¬ 
ment  was  signed,  essentially  at  the  end  of  the  TD  phase. 
The  impact  of  this  "gap"  was  an  additional  expenditure 
of  funds  to  extend  the  TD  contracts  and,  as  a  result,  the 
I  program  realized  a  schedule  delay  in  starting  the  EMD 
'  phase. 

•  Workload  Issues  with  DoDI  5000.02  and  SEC- 
NAVINST  4105.1B  and  Certification  Requirements. 

The  SecNav  Independent  Logistics  Assessment 
process  required  a  review  of  supportability  plans 
prior  to  a  single  vendor  down-select  at  Milestone  B. 

This  created  a  situation  in  which  documentation  from 
vendors  could  not  be  provided  to  assessors  without 
nondisclosure  agreements,  and,  in  some  cases,  spe¬ 
cial  training  due  to  the  nature  of  source  selection  sensi¬ 
tive  materials.  This  resulted  in  an  overarching  Life  Cycle 
Sustainment  Plan  being  evaluated  and  not  the  initial 
product  support  strategies  from  the  vendors,  in  effect 
tripling  the  workload  of  the  logistics  team  in  this  area. 


milestone  decision,  and  build  the  RFP  for  the  follow-on  for 
the  EMD  acquisition  phase.  The  net  effect  of  competitive 
prototyping  was  a  large  increase  in  workload  for  the  JATAS 
Program  Management  Office  team. 

•  Reduced  Government  Influence  on  JATAS  Design:  As  a  by¬ 
product  of  the  competitive  environment  in  the  TD  phase,  the 
government  played  a  limited  role  in  influencing  the  JATAS 
engineering  design.  To  avoid  "technical  leveling"  between 
the  two  TD  contractors,  the  government  team  restricted 
itself  to  (1)  ensuring  the  contractors  fully  understood  the 
government's  requirements  and  intent,  (2)  ensuring  the 
government  fully  understood  each  contractor's  approach 
to  meeting  the  requirements,  and  (3)  providing  guidance 
about  perceived  risk  if  a  particular  approach  might  fall  short 
of  government  requirements.  Beyond  that,  the  government 
explicitly  did  not  provide  specific  technical  direction  or  de¬ 
sign  solutions  to  the  TD  contractors.  As  a  result,  the  gov¬ 
ernment's  ability  to  influence  the  JATAS  design  early  in  its 
development  was  limited.  Similarly,  the  government  was 
limited  in  its  ability  to  adopt  good  concepts  from  either  TD 
contractor  into  its  requirements  because  the  JATAS  specifi¬ 
cation  could  not  be  changed  for  the  benefit  of  one  contractor 
over  the  other. 

•  Timing  Issues  With  Gate  Review  Process/  Contract  Gap. 

The  Secretary  of  the  Navy  (SecNav)  Gate  Review  Process 
(see  Figure  1  on  p.  21)  did  not  optimally  align  with  the  TD 
competitive  prototyping  strategy  as  it  did  not  allow  execu¬ 
tion  of  the  EMD  source  selection  in  parallel  with  execution 
of  the  TD  contracts.  The  result  was  a  gap  between  the  end 
of  the  TD  period  of  performance  and  award  of  the  EMD 
contract  because  the  RFP  for  the  EMD  source  selection 
could  not  be  released  until  Preliminary  Design  Reviews 


In  addition,  the  timing  of  the  Independent  Logistics  As¬ 
sessment  was  too  late  in  the  process  to  be  truly  effective. 
To  restore  a  proactive  stance  to  the  assessment,  it  should 
be  conducted  prior  to  releasing  the  RFP  instead  of  a  pre¬ 
scribed  length  of  time  prior  to  the  milestone.  Focusing  only 
on  the  milestone  resulted  in  passing  the  window  of  oppor¬ 
tunity  to  influence  the  Statement  of  Work,  Contract  Data 
Requirements  Lists,  and  all  other  associated  deliverables 
(and  their  timing). 

Lessons  Learned 

After  looking  at  the  positives  and  challenges  associated  with 

PMA-272's  competitive  prototyping  during  the  TD  phase,  sev¬ 
eral  lessons  were  learned  from  this  experience. 

•  The  program  did  not  have  an  established  Acquisition  Pro¬ 
gram  Baseline  when  the  competitive  prototyping  policy 
guidance  was  released.  Upfront  and  early  communication 
with  Navy  and  Office  of  the  Secretary  of  Defense  staff  iden¬ 
tified  that  this  emerging  policy  would  be  applicable  to  this 
program.  Close  collaboration  with  the  resource  sponsor  and 
policy  authorities  allowed  the  program  to  define  a  TD  strat¬ 
egy  intended  to  meet  the  needs  of  all  stakeholders.  Staying 
abreast,  and,  in  some  cases,  ahead  of  emerging  policy  guid¬ 
ance  can  help  a  program  team  make  progress  in  a  changing 
environment. 

•  The  standard  process  requirements  of  maturing  a  system 
through  Preliminary  Design  Review  will  limit  the  number  of 
contractor  teams  that  can  be  effectively  managed  during 
competitive  prototyping. 

•  The  complexity  and  number  of  the  system  interfaces  (inter¬ 
nal  and  external)  required  to  successfully  field  a  system  will 
limit  the  number  of  contractor  teams  that  can  be  effectively 
managed  during  competitive  prototyping. 
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Figure  1.  Navy  Gate  Review  Alignment  with  DoDI  5000.02  Phases  and  Milestones 
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•  Competitive  prototyping  requires  the  government  team 
to  practice  thoughtful  engagement  when  dealing  with  the 
contractor  team.  The  government  team  member  must  stop 
and  think  of  the  competitive  environment  before  directing 
or  responding  to  the  contractor.  This  requires  the  govern¬ 
ment  team  member  to  look  at  all  decisions  from  multiple 
viewpoints.  Hopefully,  this  will  result  in  a  more  thoughtful 
response. 

•  The  JATAS  team  felt  that  a  single  government  team  working 
to  ensure  its  direction  would  not  result  in  technical  leveling, 
but  would  provide  better  and  more  balanced  leadership  than 
two  government  teams  with  strict  firewalls. 

•  It  is  important  to  make  sure  the  team  agrees  on  how  the 
competition  will  be  conducted.  The  contracts  and  techni¬ 
cal  teams  may  have  different  views  regarding  how  best  to 
maintain  a  "level  playing  field."  To  the  program  manager, 
procuring  contracting  officer,  and  technical  team,  play¬ 
ing  "fair"  may  consist  of  providing  both  teams  the  same 
information. 

•  When  the  program  is  to  down-select  from  competitors, 
plan  for  the  gap  that  will  fall  between  the  final  demon¬ 
stration/presentation  and  award  of  the  follow-on  contract. 
JATAS  chose  to  have  both  teams  continue  on  risk-reduc¬ 
tion  efforts. 

•  There  is  a  price  for  competitive  programming,  and  the  pro¬ 
gram  will  need  sufficient  funding. 

•  The  maturity  of  the  system  specification  going  into  a  com¬ 
petitive  prototyping  environment  is  extremely  important 
as  there  potentially  will  be  at  least  twice  the  number  of 
requests  for  clarification.  This  increased  workload  will  only 
further  stretch  already  limited  program  resources. 


•  The  user  community  must  be  involved  early  and  invested 
in  helping  create  the  system  the  warfighters  need.  The 
platform  office  provides  the  detailed  information  regarding 
the  environment  in  which  the  system  will  be  operated  and 
maintained  once  it  has  been  installed  successfully. 

•  Logistics  and  Test  and  Evaluation  are  part  of  the  technical 
team  and  should  be  invited  to  the  engineering  meetings 
whenever  possible. 

•  When  a  program  uses  competitive  prototyping  during  the 
TD  phase  and  the  EMD  contract  is  to  be  Fixed-Price  Incen¬ 
tive-Firm,  it  can  be  difficult  for  the  government  team  to  have 
a  meaningful  dialogue  with  the  contractor  and  provide  tech¬ 
nical  direction  and  insight  without  causing  "technical  level¬ 
ing"  during  TD  or  out-of-scope  requirements  during  EMD. 

Summary 

Though  the  final  chapter  is  yet  to  be  written,  the  AN/AAR- 
59  competitive  prototyping  effort  appears  to  have  been  a 
technical  success.  But  the  savings  and/or  costs  associated 
with  resources  and  schedule  are  still  unknown.  High-risk  TD 
programs  should  consider  this  prototyping  strategy  as  a  risk 
mitigation  strategy.  In  addition,  the  competitive  nature  of  the 
contracts  forces  the  contractors  to  be  responsive  to  cost, 
schedule,  and  performance  during  the  system's  development. 

It  is  hoped  that  the  sharing  of  PMA-272's  competitive  prototyp¬ 
ing  positives  and  challenges  may  help  future  program  manage¬ 
ment  teams  as  they  make  a  determination  to  include  competi¬ 
tive  prototyping  in  their  acquisition  strategy,  and  that  the  lessons 
learned  can  add  to  the  proficiency  of  the  process.  ^ 


The  authors  can  be  contacted  at  paul.overstreet@)jsf.mil, 
Duane.Mallicoat@dau.mil,  and  Bradley.Bates(S)dau.mil. 
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